{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create TensorFlow Wide and Deep Model\n",
    "\n",
    "**Learning Objective**\n",
    "- Create a Wide and Deep model using the high-level Estimator API \n",
    "- Determine which features to use as wide columns and which to use as deep columns\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook, we'll explore modeling our data using a Wide & Deep Neural Network. As before, we can do this uisng the high-level Estimator API in Tensorflow. Have a look at the various other models available through the Estimator API in [the documentation here](https://www.tensorflow.org/api_docs/python/tf/estimator). In particular, have a look at the implementation for Wide & Deep models. \n",
    "\n",
    "Start by setting the environment variables related to your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PROJECT = \"cloud-training-demos\"  # Replace with your PROJECT\n",
    "BUCKET = \"cloud-training-bucket\"  # Replace with your BUCKET\n",
    "REGION = \"us-central1\"            # Choose an available region for Cloud MLE\n",
    "TFVERSION = \"1.14\"                # TF version for CMLE to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"BUCKET\"] = BUCKET\n",
    "os.environ[\"PROJECT\"] = PROJECT\n",
    "os.environ[\"REGION\"] = REGION\n",
    "os.environ[\"TFVERSION\"] = TFVERSION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "if ! gcloud storage ls | grep -q gs://${BUCKET}/; then\n",
    "    gcloud storage buckets create --location=${REGION} gs://${BUCKET}\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the csv files we created in the previous notebooks that we will use for training/eval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls *.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create TensorFlow model using TensorFlow's Estimator API ##\n",
    "\n",
    "We'll begin by writing an input function to read the data and define the csv column names and label column. We'll also set the default csv column values and set the number of training steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 1**\n",
    "\n",
    "To begin creating out Tensorflow model, we need to set up variables that determine the csv column values, the label column and the key column. Fill in the TODOs below to set these variables. Note, `CSV_COLUMNS` should be a list and `LABEL_COLUMN` should be a string. It is important to get the column names in the correct order as they appear in the csv train/eval/test sets. If necessary, look back at the previous notebooks at how these csv files were created to ensure you have the correct ordering. \n",
    "\n",
    "We also need to set `DEFAULTS` for each of the CSV column values we prescribe. This will also the a list of entities that will vary depending on the data type of the csv column value. Have a look back at the previous examples to ensure you have the proper formatting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Determine CSV, label, and key columns\n",
    "CSV_COLUMNS = # TODO: Your code goes here\n",
    "LABEL_COLUMN = # TODO: Your code goes here\n",
    "\n",
    "# Set default values for each CSV column\n",
    "DEFAULTS = # TODO: Your code goes here\n",
    "TRAIN_STEPS = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the input function\n",
    "\n",
    "Now we are ready to create an input function using the Dataset API.\n",
    "\n",
    "#### **Exercise 2**\n",
    "\n",
    "In the code below you are asked to complete the TODOs to create the input function for our model. Look back at the previous examples we have completed if you need a hint as to how to complete the missing fields below. \n",
    "\n",
    "In the first block of TODOs, your `decode_csv` file should return a dictionary called `features` and a value `label`.\n",
    "\n",
    "In the next TODO, use `tf.gfile.Glob` to create a list of files that match the given `filename_pattern`. Have a look at the documentation for `tf.gfile.Glob` if you get stuck.\n",
    "\n",
    "In the next TODO, use `tf.data.TextLineDataset` to read text file and apply the `decode_csv` function you created above to parse each row example. \n",
    "\n",
    "In the next TODO you are asked to set up the dataset depending on whether you are in `TRAIN` mode or not. (**Hint**: Use `tf.estimator.ModeKeys.TRAIN`). When in `TRAIN` mode, set the appropriate number of epochs and shuffle the data accordingly. When not in `TRAIN` mode, you will use a different number of epochs and there is no need to shuffle the data. \n",
    "\n",
    "Finally, in the last TODO, collect the operations you set up above to produce the final `dataset` we'll use to feed data into our model. \n",
    "\n",
    "Have a look at the examples we did in the previous notebooks if you need inspiration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create an input function reading a file using the Dataset API\n",
    "# Then provide the results to the Estimator API\n",
    "def read_dataset(filename_pattern, mode, batch_size = 512):\n",
    "    def _input_fn():\n",
    "        def decode_csv(line_of_text):\n",
    "            columns = # TODO: Your code goes here\n",
    "            features = # TODO: Your code goes here\n",
    "            label = # TODO: Your code goes here\n",
    "            return features, label\n",
    "    \n",
    "        # Create list of files that match pattern\n",
    "        file_list = # TODO: Your code goes here\n",
    "\n",
    "        # Create dataset from file list\n",
    "        dataset = # TODO: Your code goes here\n",
    "\n",
    "        # In training mode, shuffle the dataset and repeat indefinitely\n",
    "        # TODO: Your code goes here\n",
    "\n",
    "        # This will now return batches of features, label\n",
    "        dataset = # TODO: Your code goes here\n",
    "        return dataset\n",
    "    return _input_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the feature columns\n",
    "\n",
    "Next, define the feature columns. For a wide and deep model, we need to determine which features we will use as wide features and which to pass as deep features. The function `get_wide_deep` below will return a tuple containing the wide feature columns and deep feature columns. Have a look at [this blog post on wide and deep models](https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html) to remind yourself how best to describe the features.\n",
    "\n",
    "#### **Exercise 3**\n",
    "\n",
    "There are different ways to set up the feature columns for our Wide & Deep model. In the cell below, we create a function called `get_wide_deep`. It has no arguments but returns a tuple containing two things: a list of our wide feature columns, and a list of our deep feature columns.\n",
    "\n",
    "- In the first block of TODOs below, you are asked to create a list and assing the necessary feature columns for the features `is_male`, `plurality`, `mother_age` and `gestation_weeks`. Think about the nature of these features and make sure you use the appropriate `tf.feature_column`.\n",
    "- In the next TODO, you will create the bucketized features for `mother_age` and `gestation_weeks`. Think about a values to set for the `boundaries` argument for these feature columns. **Hint**: use `np.arange([start],[stop],[step]).tolist()` to easily create boundaries.\n",
    "- Next, create a list of the appropriate feature columns you created above to define the `wide` columns of our model.\n",
    "- In the next two TODOs, create a crossed feature column that uses all of the `wide` columns you created above. You'll want to use a large enough `hash_bucket_size` to ensure there aren't collisions. Then, use that crossed feature column to create a feature column that embeds `fc_crossed` into a lower dimension.\n",
    "- Finally, collect the deep feature columns you created into a single list called `deep`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_wide_deep():\n",
    "    # Define column types\n",
    "    fc_is_male,fc_plurality,fc_mother_age,fc_gestation_weeks = [# TODO: Your code goes here]\n",
    "\n",
    "    # Bucketized columns\n",
    "    fc_age_buckets = # TODO: Your code goes here\n",
    "    fc_gestation_buckets = # TODO: Your code goes here\n",
    "\n",
    "    # Sparse columns are wide, have a linear relationship with the output\n",
    "    wide = [# TODO: Your code goes here]\n",
    "\n",
    "    # Feature cross all the wide columns and embed into a lower dimension\n",
    "    fc_crossed = # TODO: Your code goes here\n",
    "    fc_embed = # TODO: Your code goes here\n",
    "\n",
    "    # Continuous columns are deep, have a complex relationship with the output\n",
    "    deep = [# TODO: Your code goes here]\n",
    "\n",
    "    return wide, deep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Serving Input function \n",
    "\n",
    "To predict with the TensorFlow model, we also need a serving input function. This will allow us to serve prediction later using the predetermined inputs. We will want all the inputs from our user.\n",
    "\n",
    "#### **Exercise 4**\n",
    "In the first TODO below, create the `feature_placeholders` dictionary by setting up the placeholders for each of the features we will use in our model. Look at the documentation for `tf.placeholder` to make sure you provide all the necessary arguments. You'll need to create placeholders for the features\n",
    "- 'is_male'\n",
    "- 'mother_age'\n",
    "- 'plurality'\n",
    "- 'gestation_weeks'\n",
    "- 'key'\n",
    "\n",
    "You'll also need to create the features dictionary to pass to the `tf.estimator.export.ServingInputReceiver` function. The `features` dictionary will reference the `fearture_placeholders` dict you created above. Remember to expand the dimensions of the tensors you'll incoude in the `features` dictionary to accomodate for batched data we'll send to the model for predicitons later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def serving_input_fn():\n",
    "    feature_placeholders = # TODO: Your code goes here\n",
    "    \n",
    "    features = # TODO: Your code goes here\n",
    "    \n",
    "    return tf.estimator.export.ServingInputReceiver(features = features, receiver_tensors = feature_placeholders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the model and run training and evaluation\n",
    "\n",
    "Lastly, we'll create the estimator to train and evaluate. In the cell below, we'll set up a `DNNRegressor` estimator and the train and evaluation operations. \n",
    "\n",
    "#### **Exercise 5**\n",
    "\n",
    "In the cell below, complete the TODOs to create our model for training. \n",
    "- First you must create your estimator using `tf.estimator.DNNLinearCombinedRegressor`. \n",
    "- Next, complete the code to set up your `tf.estimator.TrainSpec`, selecting the appropriate input function and dataset to use to read data to your function during training. \n",
    "- Next, set up your `exporter` and `tf.estimator.EvalSpec`.\n",
    "- Finally, pass the variables you created above to call `tf.estimator.train_and_evaluate`\n",
    "\n",
    "Be sure to check the documentation for these Tensorflow operations to make sure you set things up correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_and_evaluate(output_dir):\n",
    "    EVAL_INTERVAL = 300\n",
    "    run_config = tf.estimator.RunConfig(\n",
    "        save_checkpoints_secs = EVAL_INTERVAL,\n",
    "        keep_checkpoint_max = 3)\n",
    "\n",
    "    estimator = # TODO: Your code goes here\n",
    "    train_spec = # TODO: Your code goes here\n",
    "    exporter = # TODO: Your code goes here\n",
    "    eval_spec = # TODO: Your code goes here\n",
    "\n",
    "    tf.estimator.train_and_evaluate(# TODO: Your code goes here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Run the model\n",
    "shutil.rmtree(path = \"babyweight_trained_wd\", ignore_errors = True) # start fresh each time\n",
    "train_and_evaluate(\"babyweight_trained_wd\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the results of your training job above. What RMSE (`average_loss`) did you get for the final eval step?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2017-2018 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
